HEWLETT-PACKARD COMPANY 

Jntellectual Property Administration" 

sp. Box 272400 
F^J^Collins, Colorado 80527-2400 



PATENT APPLICATION 

ATTORNEY DOCKET NO. 1 0006286-1 



nun • r 



1 3 2MB « 

^ntor(s): uilas GARGI et al 
Application No.: 09/904,627 
Filing Date: j u |y 16, 2001 
Title: 



IN THE 

UNITED STATES PATENT AND TRADEMARK OFFICE 



Hierarchical Image Feature- 
Based Visualization 



Commissioner for Patents 
PO Box 1450 

Alexandria, VA 22313-1450 



Confirmation No.: 2970 
Examiner: Jin Cheng Wang 
Group Art Unit: 2672 

RECEIVED 

NOV 1 4 2003 

Technology Center 2600 



TRANSMITTAL LETTER FOR RESPONSE/AMENDMENT 



Sir: 



Transmitted herewith is/are the following in the above-identified application: 

(X) Response/Amendment ( ) Petition to extend time to respond 

( ) New fee as calculated below ( ) Supplemental Declaration 

( ) No additional fee (Address envelope to "Mail Stop Non-Fee Amendment") 

( ) Other: (fee $ ) 



CLAIMS AS AMENDED BY OTHER THAN A SMALL ENTITY 


(1) 
FOR 


(2) 

CLAIMS REMAINING 
AFTER AMENDMENT 


(3) 
NUMBER 
EXTRA 


(4) 

HIGHEST NUMBER 
PREVIOUSLY PAID FOR 


(5) 
PRESENT 
EXTRA 


(6) 
RATE 


(7) 

ADDITIONAL 
FEES 


TOTAL 
CLAIMS 




MINUS 




0 


X $18 


$ 0 


INDEP. 
CLAIMS 




MINUS 




0 


x $86 


$ 0 


[ ] FIRST PRESENTATION OF A MULTIPLE DEPENDENT CLAIM 


+ $290 


$ 0 


EXTENSION 
FEE 


1ST MONTH 


2ND MONTH 


3RD MONTH 


4TH MONTH 


$ 0 


$1 10.00 




5420:00 




$950.00 




51480.00 




OTHER FEES 


$ 


TOTAL ADDITIONAL FEE FOR THIS AMENDMENT 


$ 0 



Charge $ g to Deposit Account 08-2025. At any time during the pendency of this 

application, please charge any fees required or credit any overpayment to Deposit Account 08-2025 
pursuant to 37 CFR 1.25. Additionally please charge any fees to Deposit Account 08-2025 under 37 
CFR 1.16 through 1.21 inclusive, and any other sections in Title 37 of the Code of Federal Regulations 
that may regulate fees. A duplicate copy of this sheet is enclosed. 

Respectfully submitted, 



Date: 



Ullas 



I hereby certify that. this document is being filed by personal 
delivery to the Customer Service Window, Crystal Plaza 2, 2011 
South Clark Place, Arlington Virginia, of the United States Patent & 
Tradem^rjc Office on the date indicated above. 



By 




RGI et al 





ignature and Reg 



:ated at 
No/ 



Patrick C. Keane 



Attorney/Agent for Applicant(s) 
Reg. No. 32,858 

Date: Nov. 13, 2003 



Rev 10/03 (TransAmd) 



Telephone No.: (703) 838-6522 

Attach as First Page to Transmitted Papers - 




Patent 

Attorney's Docket No. 10006286-1 
IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 



In re Patent Application of 

Ullas Gargi et al 

Application No . : 09/904 , 627 

Filed: July 16, 2001 

For: HIERARCHICAL IMAGE 
FEATURE-BASED 
VISUALIZATION 



Group Art Unit: 2672 
Examiner: Jin Cheng Wang 
Confirmation No.: 2970 

RECEIVFD 

NOV 1 4 2003 

Technology Center 2600 



REQUEST FOR RECONSIDERATION 

Commissioner for Patents 

P.O. Box 1450 

Alexandria, VA 22313-1450 

Sir: 

In response to the Office Action mailed August 18, 2003, reconsideration and 
allowance of the present application are respectfully requested. Claims 1-20 remain 
pending in the application. 

In numbered paragraph 3 on page 2 of the Office Action, claims 1-8, 11-14 and 16- 
19 are rejected under 35 U.S.C. §102(e) as being anticipated by U.S. Patent No. 
6,584,221 (Moghaddam et al). In paragraph number 7, claims 9-10 and 20 are rejected 
under 35 U.S.C. §103(a) as being unpatentable over the Moghaddam patent in combination 
with U.S. Patent No. 6,121,969 (Jain). In numbered paragraph 9, claim 15 is rejected 
under 35 U.S.C. §103(a) as being unpatentable over the Moghaddam patent in combination 
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with U.S. Patent No. 5,528,259 (Bates). These rejections are respectfully traversed, as the 
documents relied upon by the Examiner fail to teach or suggest Applicants' invention as set 
forth in independent claims 1 and 17. 

The present invention is directed to a multimedia database and classification system 
which can, for example, provide for automatic classification and retrieval of multimedia 
files based on features of the multimedia files. Figure 1 shows an exemplary method for 
visualizing and retrieving data files in accordance with an exemplary embodiment of the 
present invention. In step 110, a plurality of images representing data files on a display 
device are displayed using a first (i.e., coarse) distance metric. The distance metric 
represents a distance between each data file. In step 120, a portion of the images can be 
redisplayed on the display device using a refined distance metric. As described on 
specification page 6 beginning with line 4, the distance metric can be refined at each of 
plural redisplays until a desired data file is found or a maximum refined distance metric is 
reached. In step 140, a desired data file can be retrieved, and/or can be marked or 
selected. 

Figures 2A-2E show a graphical representation of displays generated in accordance 
with exemplary embodiments, where each portion of the images redisplayed can be 
graphically selected by a user. Figure 2 A is a screen capture of an exemplary display 
showing a first level of images representing data files on a two-dimensional display. A 
first, coarse, distance metric can be calculated, and allows the user to receive useful 
information about the organization of images on the display. Based on perceived 
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properties of a desired image, a user can select an area 202 of the screen where a desired 
image most likely resides. Figure 2B shows a portion of images redisplayed as selected by 
area 202. Here the distance metric has been recalculated using more of the image 
information than was used in the first distance calculation. As described on specification 
page 9, the reclustering and redisplaying of selected images is more than merely a zoom 
function. 

The redisplayed process can be repeated, as represented by Figures 2C and 2D. 
Distance metrics between images are again recalculated using more image feature data than 
previously used with respect to Figure 2B. Following the redisplay based on recalculated 
distance metrics, a desired data file (e.g., image 240) is identifiable and can be selected and 
retrieved as shown in Figure 2E. Figure 3 shows another exemplary method which can be 
implemented in accordance with the present invention, and Figure 4 shows a flowchart for 
performing a coarse to fine distance calculation. 

The foregoing features are broadly encompassed by independent claims 1 and 17. 
Exemplary embodiments of the present invention display images using a first distance 
metric between each data file, and provide for redisplaying a portion of the images using 
a refined distance metric in a manner which constitutes more than merely a zoom feature. 
Claim 1 is directed to a method of visualizing and retrieving a data file from a second data 
file. The claim 1 method includes a step of displaying a plurality of images representing 
corresponding data files on a display device using a first distance metric between each 
data file. Claim 1 also recites also redisplaying a portion of the images on the display 
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device using a refined distance method. Claim 17 is directed to a method of interactively 
retrieving a data file from a set of data files in real time, and includes features similar to 
those mentioned with respect to claim 1 . 

The foregoing features are neither taught nor suggested by the Moghaddam patent, 
considered individually or in combination with Jain or Bates patents in the manner 
suggested by the Examiner. 

The primary reference relied upon by the Examiner, the Moghaddam patent, is at 
best, an image retrieval method which provides a zoom feature. The method described by 
Moghaddam separates and filters images to extract color and texture features. These 
features for each image are partitioned in a plurality of blocks. A joint distribution of the 
color features and a joint distribution of the texture features are estimated for each block, 
and stored in a database with each image to enable retrieval of images by comparing the 
estimated joint distribution. Column 2, lines 59-67 of the Moghaddam patent describe 
using color and texture features as key image components for image retrieval. Based on 
the color/texture features, images are retrieved in response to a user query and are 
presumably displayed in rank order. That is, retrieved images are presumably displayed 
based on their proximity to the user's query, as there is no teaching or suggestion of 
displaying images based on a distance metric between each data file. Images in the 
Moghaddam database are displayed with respect to a particular query, and not with respect 
to one another. Accordingly, the response to any request to redisplay a portion of the 
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original set of displayed images would be based on the original distances to the original 
query, and not on a refined distance metric. 

This distinction can be characterized by way of example. With exemplary 
embodiments of the present invention, when a particular image is removed from the set of 
images to be displayed, all of the distance metrics between images will change resulting in 
relocation of the remaining images on the display. In contrast, when is a subset of images 
displayed in response to a query is selected for redisplay, the relationship among the 
images in the redisplay will remain the same as that of the original display. This is 
because in the Moghaddam system, all images are displayed with respect to an initial 
query, and not with respect to distance metrics between each data file to be displayed. The 
Moghaddam redisplay function is simply directed to a zoom feature. 

As such, claims 1 and 17 are allowable over the Moghaddam patent. The 
remaining patents to Jain and Bates fail to overcome the deficiencies of the Moghaddam 
patent, such that claims 1 and 17 are considered allowable. The Jain patent is directed to 
navigating to re-rank images. This patent is not directed to using a distance metric 
between data files, or to redisplaying a portion of the images on the display device using a 
refined distance metric. 

The Bates patent is directed to visualization of images, and is not directed to 
computing any distance metric whatsoever. As such, claims 1 and 17 are considered 
allowable over the Moghaddam patent even when this patent is considered in combination 
with the Jain and Bates patent in the manner suggested by the Examiner. 
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Because independent claims 1 and 17 are considered allowable, all of the remaining 
claims 2-16 and 18-20 are considered allowable. 

All rejections and objections raised in the Office Action having been addressed, it is 
respectfully submitted that the present application is in condition for allowance and a 
Notice of Allowance is respectfully solicited. 



Respectfully submitted, 



Burns, Doane, Swecker & Mathis, L.L.P. 
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